The D2Q2 Framework: On the Relationship and Combination of Language Modelling and TF-IDF

Language Modelling (LM) and TF-IDF are two retrieval models with different foundations. There have been efforts aiming at establishing the relationship between these models, and whether one includes the other. Whether their combination could yield a third and better model is an open research question. This paper revisits the foundations of LM and TF-IDF and explores how these models’ bare structures relate and how these structures can be combined. We begin with the premise that TF-IDF is the P (d|q)/P (d) side of retrieval, which complements the common view that LM is P (q|d)/P (q). Next, a hybrid framework based on the decomposition of the product of the two sides, P (d|q)/P (d) · P (q|d)/P (q), is developed. This leads to the D2Q2 family of models, which joins the inner components of LM and TF-IDF instead of combining their scores. This paper provides new insights into the relationship between LM and TF-IDF, and experimental results show that the D2Q2 models perform comparably to competitive baselines.